DESCRIBING WEST-3-STACK-SORTABLE PERMUTATIONS WITH 

PERMUTATION PATTERNS 



HENNING ULFARSSON 



Abstract. We describe a new method for finding patterns in permutations that produce 
a given pattern after the permutation has been passed once through a stack. We use this 
method to describe West-3-stack-sortable permutations, that is, permutations that are 
sorted by three passes through a stack. We also show how the method can be appUed 
to the bubble-sort operator. The method requires the use of mesh patterns, introduced 
by Branden and Claesson (2011), as well as a new type of generalized pattern we call a 
decorated pattern. 



Contents 

1. Introduction [l| 

2. Generalized permutation patterns S 

3. Finding preimages of patterns 

4. Describing West-3-stack sortable permutations [ll| 



5. Open problems |18 



References 19 



1. Introduction 

A permutation is a one-to-one correspondence from a finite set {l,...,n} to itself. 
Permutations will be written in one-line notation, so the permutation 2314 maps 1 to 2, 
2 to 3 and so on. The number of letters in a permutation will be called its length and the 
set of all permutations of length n is denoted Sn- The identity permutation 12- ■ - n will 
be denoted id„, or just id if n is understood from the context, or irrelevant. For integers 
a < b we use |a, bj to denote the set {a, a + 1, ... , b}. 

In the 1970's Knuth [12] initiated the study of sorting and pattern avoidance in per- 
mutations. He considered the problem of sorting a permutation by passing it through a 
stack. A stack is a structure that can store elements from a permutation. We can push 
onto the top of the stack and elements are popped out to the output. Consider trying to 
sort the permutation 231 by a stack, as shown in Figure [H 
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Figure 1. Trying, and failing, to sort 231 with a stack. The figure is read 
from right to left 



Note that we always want the elements in the stack to be increasing, from the top, 
since otherwise it would be impossible for the output to be sorted. We failed to sort the 
permutation in one pass through the stack and therefore say that it is not stack- sortahle. 
Knuth [12j showed that a permutation is stack-sortable if and only if it avoids 231 as a 
pattern. We will reprove this below in Theorem 13. II Several variations on Knuth's original 
problem have been considered, such as changing the way the stack operates, and/or 
adding more sorting devices, see Bona [5] for a survey. In this paper we will consider two 
variations: repeatedly passing a permutation through a stack and the so-called bubble- 
sort operator. We introduce a new method for finding patterns in a permutation that 
will cause these sorting devices to output a given pattern. If the given pattern we want 
to be outputted is a classical pattern (defined below) we show that the mesh patterns 
introduced by Branden and Claesson suffice, but if the given pattern is itself a mesh 
pattern we will need to introduce a new kind of generalized pattern we call a decorated 
pattern. 

In the next section we review some literature on generalized permutation patterns and 
introduce a new generalization. 

2. Generalized permutation patterns 

A standardization of a list of numbers is another list of the same length such that the 
smallest letter in the original list has been replaced with 1, the second smallest with 2, 
and so on. The standardization of 5371 is 3241. A classical {permutation) pattern is 
a permutation pm.Sk- A permutation vr in Sn contains, or has an occurrence of, the 
pattern p if there are indices 1 < «i < ■ ■ ■ < ife < n such that the standardization of 
7r(ii) • • •7r(ifc) equals the pattern p. If a permutation does not contain a pattern p we say 
that it avoids the pattern p. The permutation tt = 526413 contains the pattern p = 132, 
and has three occurrences of it, given by the subsequences 264, 263 and 243. We can 
draw the graph of the permutation by graphing the coordinates (i,7r(z)) on a grid. For 
example the permutation tt above is shown in Figure |2] where we have additionally circled 
the occurrences of the pattern p. 
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Figure 2. The permutation 526413 and three occurrences of the pattern 132 
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The same permutation avoids the pattern 123, since we can not find an increasing 
subsequence of length three in it. 

Classical patterns form the base of a hierarchy of generalizations, shown in Figure |3l 
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Figure 3. The hierarchy of generalizations of classical patterns 



We will describe in detail the mesh, marked mesh and decorated patterns, as well as 
barred patterns, as these will be the generalizations we need. We describe the others 
briefly below and give references for them. 

2.1. Mesh patterns and barred patterns. Mesh patterns were introduced in [7]. We 
review them via an example. The mesh pattern 



occurs in a permutation if we can find the underlying classical pattern 132 positioned in 
such a way that the shaded regions are not occupied by other entries in the permutation. 
Consider the permutation 526413. From above we know that the classical pattern has 
three occurrences in this permutation. In Figure H] one can see that just one of these 
satisfies the additional requirement that there be no additional entries in the shaded 
region "between and to the left of the 3 and the 2" . 

Another way of writing a mesh pattern is to give the underlying classical pattern, 
followed by the set of shaded boxes, labelled by their lower left corner (the left-most 
box in the bottom-most row being (0,0)). The mesh pattern we considered here is 
(132, {(0, 2), (1, 2), (2, 2)}) and any classical pattern p can be written as the mesh pat- 
tern (p, 0). 
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Figure 4. The permutation 526413 and one occurrence of the mesh pat- 
tern Q above 
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Figure 5. The mesh pattern (132, {(0, 2), (1, 2), (2, 2)}) 



Describing the simple permutations with mesh patterns. Recall that an interval in a per- 
mutation is a set of entries that is consecutive in positions and values. An interval is 
trivial if consists of one letter, or is the entire permutation. The elements 4653 form a 
nontrivial interval in the permutation 28465317. 
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A permutation without nontrivial intervals is called simple. Simple permutations have 
been shown to be useful in the study of classical pattern classes, see e.g., Albert and 
Atkinson [1] and Brignall [8]. 

Notice that the left-most, right-most, highest and lowest elements (hereafter called the 
boundary elements) in the interval 4653 form an occurrence of the mesh pattern 




This leads to the following proposition. 

Proposition 2.1. A permutation is simple if and only if it avoids each of the patterns 
below, as well as their symmetries generated by the symmetries of the square. 
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Proof. A permutation is not simple if and only if it has a non-trivial interval, and any 
interval must have boundary elements satisfying one of the mesh patterns above, or their 
symmetries. The element in the patterns that is not one of the boundary elements is 
added to ensure that the interval is not the entire permutation, and thus trivial. □ 

Barred patterns were introduced by West [IT]. A barred pattern is a classical pattern 
with bars over some of the entries. Such a pattern is contained in a permutation if the 
standardization of the unbarred entries is contained in the permutation in such a way that 
they are not part of an occurrence of the whole barred pattern. This is best explained 
by considering examples. The barred pattern 35241 does not occur in the permutation 
416352, since there is only one occurrence of the classical pattern we get from the unbarred 
entries, in the subsequence 4352, and this occurrence is part of an occurrence of 35241. 
The permutation 5264173 does contain this barred pattern, in the subsequence 5473, since 
that is an occurrence of 3241 that is not part of an occurrence of 35241. In [7] it was 
shown that any barred pattern with one barred entry is a mesh pattern. The barred 
pattern we discussed here is in fact the mesh pattern 
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This explains the edge from barred patterns with one bar ("1-barred") to mesh patterns 
( "mesh" ) in Figure [31 

2.2. Marked mesh patterns. Marked mesh patterns were introduced by the author in 
[15] and used in joint work with Woo in [16] in the characterization of local complete 
intersection Schubert varieties. They give finer control over whether a certain region in a 
permutation is allowed to contain elements, and if so, how many. Again we just give an 
example. Consider the marked mesh pattern below. 
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The meaning of the 1 in the region containing boxes (1, 0), (1, 1), (2, 0) and (2, 1) is that 
this region must contain at least one entry. In Figure |6] we see that there is exactly one 
occurrence of this mesh pattern in the permutation 526413. 




Figure 6. The permutation 526413 and one occurrence of a marked mesh pattern 



Marked mesh patterns will be useful below when we need to add elements into an 
existing pattern to ensure other elements are popped out of a particular sorting device. 

2.3. Decorated patterns. Below we will need even finer control over what is allowed 
inside a particular region in a pattern. We will need to control whether the entries in the 
region avoid a particular pattern. Consider for example the decorated pattern 



The decorated region in the middle signifies that an occurrence of this pattern should 
be an occurrence of the underlying classical pattern 21 that additionally does not have 
entries in the region that contain the pattern 12 - or equivalently - whatever is in that 
region must be in descending order, from left to right. In Figure [7| there is an occurrence 
of the decorated pattern on the left and on the right we have an occurrence of the classical 
pattern 21 that does not satisfy the requirements of the decorated region. 




Figure 7. The permutation 526413 and one occurrences of a decorated pattern 



Below we state the formal definition of a decorated pattern. 

Definition 2.2. A decorated pattern {p,C) of length k consists of a classical pattern p of 
length k and a collection C which contains pairs (C, q) where C is a subset of the square 
|0, fc] X |0, kj and q is some pattern, possibly another decorated pattern. An occurrence of 
(p, C) in a permutation vr is a subset u of the diagram G{7i) = {{i, 7r(i)) \ l < i < n} such 
that there are order-preserving injections a, (3 : |l,/c] — )■ |1,?t-] satisfying two conditions: 

(1) oo = {{a{t),m)--{hj)^G{p)}. 
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(2) Let Rij = la{i) + 1, a{i + 1) - 1] x + 1, f3{j + 1) - l], with a(0) = = /3(0) 
and a{k + 1) = n + 1 = f3{k + 1). For each pair (C, q) we let C = IJ{j j)ec 
require that 

C n G{tt) avoids q. 

More bars in more places. We noted above that any barred pattern with just one barred 
entry can be translated into a mesh pattern. If there is more than one bar, on entries 
that are adjacent in the pattern, this is no longer true. But we can still translate these 
types of barred patterns into decorated patterns. For example the barred pattern 1234 is 
equivalent to the decorated pattern 

There are other barred patterns, such as 1324, that can not be expressed as decorated 
patterns. 

2.4. Other generalizations. Vincular patterns, sometimes called dashed patterns, were 
defined by Babson and Steingrfmsson [1] and are mesh patterns where shaded regions must 
be vertical strips. Bivincular patterns were defined by Bousquet- Melon, Claesson, Dukes 
and Kitaev [6], and are mesh patterns where shaded regions can be either vertical or 
horizontal strips. 

Bruhat-restricted patterns and interval patterns were defined by Yong and Woo in 
[18] and [19], and have applications to the study of singularities of Schubert varieties. 
In [T5] the author showed that interval patterns are a special case of mesh patterns. 
McGovern [13] showed that classical patterns, with the extra restriction that occurrences 
must be a union of cycles, have applications to geometry as well. It is shown in [15] that 
this generalization is subsumed by marked mesh patterns. 

Profile classes were introduced in Murphy and Vatter [13], and later called monotone 
grid classes in Huczynska and Vatter [TT]. It is not hard to show that a permutation is a 
member of a particular monotone grid class if and only if it contains one pattern from a 
finite list of decorated patterns. 

3. Finding preimages of patterns 

In this section we define a method for describing patterns that are guaranteed to produce 
a given pattern in a permutation after it is sorted by a stack or with the bubble-sort 
operator. 

3.1. The stack-sort operator. For a permutation vr we denote with ^'(vr) the image 
of vr after it is passed once through a stack. Recall that a permutation is stack- sort able 
if and only if ^(Tr) = id, the identity permutation. Another way to state this is that a 
permutation vr in Sn is stack-sortable if and only if 5'(7r) G Av„(21), where Av„(21) is 
the set of permutations of length n that avoid 21. Of course Av„(21) = {id} but framing 
the question like this leads to a generalization: Given a pattern p, what conditions need 
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to be put on vr such that 5'(7r) G Av„(p). Equivalently, we can ask for a description of 
5-i(Av„(j9)). 

Below we call permutations such that S^ln) = id, West-k- stack- sortable permutations, 
since West considered this generalization from the case of one stack first. Note that for 
k > 1 these permutations are different from the k- stack- sortable permutations, which are 
the permutations that can be sorted by using k stacks in series without the requirement 
that the permutation completely passes through one stack at a time. For example, the 
permutation 2341 is not West-2-stack-sortable, but if we put the entries 2,3,4 onto the 
first stack, pass 1 all the way to the end, and then use the second stack to sort 2, 3, 4 we 
end up with 1234. So 2341 is 2-stack-sortable. See also Open Problem (jl]) in Section \5\ 
below. 

The basic idea behind the method we are about to describe is that 5'(7r) has an occur- 
rence of a pattern p of length k if and only if the k elements making up this occurrence 
were present in vr as some kind of pattern before we sorted. This is of course a complete 
tautology but will get us pretty far. We start by showing how this idea allows us to 
describe stack-sortable permutations, as well as West-2-stack sortable permutations. 

Stack- sortable permutations. We know that vr is not sorted by the stack if and only if 
^(Tr) contains the classical pattern 21. Therefore consider a particular occurrence of 
this pattern in ^(Tr). Before sorting, the elements in this occurrence must have been an 
occurrence the pattern 



21 



in vr. In order to remain in this order the element corresponding to 2 must be popped off 
the stack by a larger element before the element corresponding to 1 enters. Thus the box 
(1, 2) must be occupied by at least one element and we have an occurrence of the marked 
mesh pattern 



which is equivalent to the classical pattern -f . We have therefore reproven Knuth's 

-trr 

result. 

Theorem 3.1 (Knuth [12j). A permutation vr is stack-sortable, i.e., S{n) = id, if and 
only if it avoids the pattern 231. 

West-2- stack- sortable permutations. We can similarly reprove West's result on West-2- 
stack sortable permutations, i.e., permutations vr such that S'^{tt) = id. By Knuth's result 
we know that vr will be sorted by two passes through the stack if and only if 5'(7r) avoids 
the pattern 231. An occurrence of 231 must have been either of the patterns 
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in IT. Consider the pattern on the left. In order for the elements to stay in this order as 
they pass through the stack we must have an element in the box (2, 3) in order to pop 
the element corresponding to 3 out of the stack before the smallest element enters. Now, 
the opposite happens for the pattern on the right. The 3 must stay on the stack until 2 
enters, so there can be no elements in the box (1,3) and then both 2 and 3 must leave 
the stack before 1 enters. Thus the patterns above become the marked mesh patterns 
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These are more naturally written as 
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The pattern on the right is equivalent to the pattern 



in the sense that a permu- 



tation either contains both patterns or avoids both, by a lemma of Hilmarsson, Jonsdottir, 
SigurSardottir, ViSarsdottir and Ulfarsson \L0\. For these two patterns it is also easy to 
see directly that they are equivalent. As mentioned above this pattern is another repre- 
sentation of the barred pattern 35241. Thus we have re-derived West's result. 

Theorem 3.2 (West [Ej). A permutation n is West-2-stack-sortable, i.e., S'^{'k) = id, if 
and only if it avoids the patterns 2341 and 35241. 

Recall that an inversion in a permutation is an occurrence of the classical pattern 21, 
while a non-inversion is an occurrence of 12. We now describe how given a classical 
pattern p the method can be used to find a set of marked mesh patterns P such that 

S-\Av{p)) = Av(P). 

There are two things to consider when choosing what classical patterns to start from. 

(1) If two elements in p are part of an inversion, they must also be part of an inversion 
in all patterns in P. 

(2) If two elements in p are part of a non-inversion, they can either be a non-inversion 
or an inversion in the patterns in P. 

We now consider what shadings or markings must be added. 

(1) If a > 6 form an inversion in a particular pattern p' in P and are supposed to come 
out of the stack as an inversion, then there must be another element c > a that 
pops the a out before b is pushed onto the stack, thus maintaining the inversion. 
If such an element is present in p' we need not do anything. If there is no such 
element we need to mark the region above a and between a and b with a "1" . 

(2) If a > 6 is an inversion in p' that must become a non-inversion in p, then we must 
make sure a stays on the stack until b arrives, and in order to do that we must 
shade all the boxes above a and between a and b. In particular there can be no 
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other elements of p' in this shaded region. If a < 6 is a non-inversion in p' then it 
will still be a non-inversion in p after sorting. 

This suffices to generate the set of marked mesh pattens P from a classical pattern p. 
We will see later that if we start with a mesh pattern p then it will be more difficult to 
find the set P and we will be forced to use the decorated patterns we introduced above. 

3.2. The bubble-sort operator. The bubble-sort operator is a very inefficient way of 
sorting a permutation. It works on adjacent locations in a permutation and if the left 
element is larger than the right one, they are swapped. For example, one pass of bubble- 
sort on the permutation 52134 produces 21345, since the 5 will travel from the front to the 
back. As another example, bubble-sorting 521634 produces 215346. A slight modification 
of the method described above works equally well for this operator. Let -B(vr) denote the 
output of one pass of bubble-sort on vr. Consider for example permutations vr such that 
B{tt) = id. We see that -B(vr) contains the pattern 21 if and only if vr contains 



To make sure that these elements stay in this order, we either need a large element in 
front of the 2, which would mean the 2 would never be moved; or we need a large element 
in between the 2 and the 1 that will stop the 2 from moving past 1. We arrive at the 
marked mesh pattern 



This pattern is equivalent to the two classical patterns 231 and 321. Albert, Atkinson, 
Bouvel, Claesson and Dukes [3] first made this result explicit, that B^n) = id if and only 
if vr avoids 231 and 321. In the same paper the authors show that for any classical pattern 
p with at least three left-to-right maxima, the third of which is not the final symbol of p, 
the set B~^{Av{p)) is not a classical pattern class, i.e., not described by classical patterns. 
We consider the smallest example of such a pattern, p = 1243 



Proposition 3.3. 



5"^(Av(1243)) = Av 
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Note that all the patterns can be expanded to mesh patterns, but we would then have 
eight patterns instead of the four above. 

Proof. It is easy to see that the underlying classical patterns are the only ones that are 
possible. Showing that the additional shadings and/or markings are necessary is similar 
for all of the patterns, so we just consider the third one. The 2 must move past the 1, 
which is achieved by adding the shading above it. The 4 must not move past the 3 so 
there must either be a large element in front of it, implying that it will never move, or 
between it and the 3, implying that it will not move past the 3. □ 
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4. Describing West-3-stack sortable permutations 



We now move on to the case of West-3-stack sortable permutations, i.e., permutations 
71 such that S^{tt) 



id. By West's result, stated in Theorem 13. 2 [ we know that vr will be 
sorted by three passes through the stack if and only if S'(7r) avoids the two patterns 
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We will use the same method as we did above, but when we consider the pattern on the 
right, the shaded box will cause some complications and the decorated patterns introduced 
above will be necessary. We therefore consider the pattern on the left first. 

Lemma 4.1. Let tt be a permutation. An occurrence of Wi in S{tt) comes from exactly 
one of the patterns below in vr. 
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It is easy to see that the patterns 231, Wi, Ii are part of a family, whose k-th member 
prohibits a permutation containing it from being West-fc-stack sortable. 

Proof. The element 5 in all the patterns is added in to pop all the large elements out 
before the 1 is pushed on the stack. It is easy to see that the underlying classical patterns 
in the lemma, without the 5, are the only possible candidates for producing an occurrence 
of Wi in S^tt). Since proving that a particular shading must be apphed to each of the 
patterns is similar, we only consider Is. This pattern comes from the elements in the 



pattern Wi having been arranged in the pattern 



in 71. In order for the elements 



to come out in the order we want, the 4 must stay on the stack until both 3 and 2 have 
been pushed onto the stack. Also, 3 must remain on the stack until 2 arrives. We must 
therefore shade boxes (1, 3), (1, 4), (2, 2), (2, 3) and (2, 4). Finally, the elements 2, 3 and 4, 
must leave the stack before 1 arrives, so we must add an extra element, the 5, in between 
4 and 1. □ 



We now consider the pattern W2, but without the shading. 
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Lemma 4.2. Let it be a permutation. An occurrence of 
exactly one of the patterns below in it. 



in S{tt) comes from 
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Proof. As in the proof of Lemma WA\ the 5 is added in to pop out the large elements and it 
is easy to see that only the classical patterns underlying ji, j^ct'^'^js (without the 5) could 
possibly produce an occurrence of 3241 in S{ti). 

(1) For the classical pattern underlying ji to become 3241 in 5'(7r) there must be an 
element in the marked region that pops the 3 out of the stack. 

(2) Boxes (2,4) and (2,5) in the classical pattern underlying j2 must be shaded to 
ensure that 3 stays on the stack until 2 is put on the stack. 

(3) The pattern js comes from the elements in the pattern W2 having been arranged 



in the pattern 



in TT. In order for the elements to come out in the order 



we want, the 4 must stay on the stack until 2 arrives and this explains the shaded 
boxes in js. Now the 3 must be popped out before the 2 arrives and that explains 
the marking in box (2, 3). □ 

We must now consider under what additional conditions the patterns ji, j2 and ja in 
the lemma will cause the correct shading in the pattern W2. We express these conditions 
in the following lemma and two propositions. 

Lemma 4.3. An occurrence of in a permutation vr will become an occurrence ofW2 in 
S{7r). 

We leave the proof to the reader. We rename the pattern J3 and note that it can also be 
expanded into a mesh pattern. 
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Proposition 4.4. An occurrence 0/^2 ^ permutation ir will become an occurrence of 
W2 in S{7t) if and only if it is part of one of the patterns below, where the elements that 
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have been added to the pattern j2 are circled. 



J2,l 



J' 



2,4 



J' 



2,10 




Proof. To ensure that there are no elements in the shaded box in W2 we must look at the 
element that pops 3 in j2- There are four different possibilities. 

mm 1 m 
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We explain the shadings and the decoration of the pattern j2,2 as the others are similar. 
For this pattern, the size of the element that popped the 3 from the stack was in-between 
the 4 and the 5. Since this was the element that popped the 3 there can be no elements 
in boxes (1, 3), . . . , (1, 6). The boxes (2, 5) and (2, 6) can not contain an element, since 
that would pop out the element we just added (the 5 in ^2,2) and this element would land 
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in the shaded box in W2. Now consider the decorated box (2, 4). It can contain elements, 
but none of them are allowed to leave the stack prior to the 4 being pushed on, since any 
one of them would then land in the shaded box in W^2- Any elements in this region must 
then be in descending order, or equivalently, avoid the pattern 12. 

We also need to make sure that there elements that arrived on the stack prior to 3 
are not popped out and into the shaded region in W2. We do the patterns in order of 
difficulty, which happens to be the reverse order. 

(1) Consider the pattern j'2,4. No additional shadings are necessary, and we rename 
this pattern J2,i2 for future reference. 

(2) Consider the pattern ^2,3- If there are elements in boxes (0,4) and (0,5) that are 
still on the stack when 3 is pushed on they will be popped by the 6 and will land 
in the shaded region in W2. We must therefore have these boxes empty, or an 
element in box (0, 6) that pops everything out before 3 is pushed on. We get the 
two patterns 




(3) Consider the pattern j2,2- If there are elements in box (0,4) that are still on the 
stack when 3 is put on they will be popped by the 5 and will land in the shaded 
region 1^2- We must therefore have this box empty, or an element in box (0, 5) or 
(0, 6) that pops everything out before 3 is pushed on. We get the three patterns 




(4) Consider the pattern j2,i- If there are elements in boxes (0, 5) and (0, 6) then they 
must not be popped by the elements in the decorated region (boxes (2, 6) and 
(2, 7)). We must therefore consider three cases: 

• The decorated region is empty: In this case we do not need to worry about 
anything prior to 3 and get the pattern 
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• The upper box of the decorated region is empty, but not the lower box: We 
choose the top-most element in the lower box and add it to the pattern. 




We can now handle this pattern the same way we handled the pattern j2,2 
above and get three patterns. 




• The upper box of the decorated region is not empty: We choose the top-most 
element and add it to the pattern. 




We can again handle this pattern the same way we handled the pattern ^2,2 
above and get two patterns. 




□ 



We now consider the last pattern, ji, and what conditions must be imposed on it in 
order to get W2 after sorting. 

Proposition 4.5. An occurrence of ji in a permutation 71 will become an occurrence of 
W2 in S{Tr) if and only if it is part of one of the patterns below, where the elements that 
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have been added to the pattern ji are circled. 
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Proof. To ensure that there are no elements in the shaded box in W2 we must look at the 
element that pops 3 in ji. There are three different possibilities. 
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We also need to make sure that elements that arrived on the stack prior to 3 are not 
popped out and into the shaded region in Wi. The derivation of the patterns in the 
proposition is identical to the proof of Proposition I4.4[ □ 



Taken together, Lemmas 14.11 and 14. 3[ with Propositions 14.41 and 14.51 produce a list of 
29 patterns describing permutations that are not West-3-stack sortable. We can simplify 
this list considerably as follows: Since the patterns Ji^i, . . . , Ji^e all imply containment of 
l\ we can remove them. Similarly J2,i2 removes Ji^, . . . , Ji ^; l\ removes J2,i, . . . , J2,6; -^2 
removes J2,7, . . . , J2,9 and J4 removes J3. 
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We are left with the fact that a permutation is West-3-stack-sortable if and only if it 
avoids the patterns 
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If the decorated region in J2,m contains an element in the lower box then it will produce 
an occurrence of If,. The same is true of J^^w- We can therefore replace these patterns 
with 




J'. 



2,10 



■^2,11 




(Keeping the names of the patterns unchanged.) The pattern Ji lo can be simplified 
as well. The lower box in the decorated region can contain at most one element, since 
otherwise we would have an occurrence of l2- We begin by replacing it with 




1. 




and notice that the second pattern above implies containment of J2,io- We therefore 
replace Ji^io with the first pattern. 
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The pattern Ji ii can be simplified in the same way. The lower box in the decorated 
region can contain at most one element, since otherwise we would have an occurrence of 
1 2- We begin by replacing it with 





and notice that the second pattern above implies containment of J2,ii. We therefore 
replace Ji n with the first pattern. 

We have therefore proven the following. 

Theorem 4.6. A permutation n is West-2)- stack- sortahle, i.e., S^{Tr) — id, if and only if 
it avoids the following decorated patterns 



h 







1 
















1— 




















— ( 


1 — 






t 


\— 


















— ( 


1 — 





1 


i 












/// 




(— 




















— ( 


1 — 








— ( 


(— 
















— ( 


1— 





m 










i " 








-» — 













, -^2,12 — 





i 


i 
































— ( 


1 — 




■^1,10 — 1 
































— 1 


•— 


















—i 





-^1,11 





) ■^2,11 



5. Open problems 

Above we only had use for decorated patterns where the entries in certain boxes needed 
to avoid a particular pattern. One might guess that in some situations it could be useful 
to require the entries to contain a particular pattern instead. Furthermore, one could go 
as far as assigning a set X(ij) to the entries in box {i,j) and then placing restrictions on 
those sets, such as -'^(i,!) = 0, #-^^(2,3) > 2, ^(0,1) avoids 123, -^^(3,2) U -^^(3,3) contains the 
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pattern 21 etc. For a concrete example, note that a fixed point is an occurrence of tlie 
pattern (l,#X(o,i) = #X(i,o))- 

We end witli some open problems related to the above. 

(1) The number of West-2-stack-sortable permutations was conjectured by West [17] 
to be 2(3n)!/((n + l)!(2n + 1)!). This conjecture was proven by Zeilberger |20j . 
Later Dulucq, Gire and West |9j found these permutations to be in bijection with 
rooted non-separable planar maps. The enumeration of West-3-stack-sortable per- 
mutations is still completely open. 

(2) One might hope the method we described above for finding a set of patterns P such 
that S~^{Av{p)) = Av(P) could be turned into an algorithm. This is definitely 
feasible for classical patterns p, but might be difficult for mesh patterns p. 

(3) Our method was shown to work as well with the bubble-sort operator. Hopefully 
it can be applied to other similar sorting operators. 

(4) We noted in subsection 13.11 that there is another set of permutations called the 2- 
stack-sortable permutations. It is known that classical patterns suffice to describe 
this set but it is not known which classical patterns. See Albert, Atkinson and 
Linton [2] for recent work on this. 

The author hopes to pursue each of the open problems mentioned above in future work. 
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